Skip to content

infra: simplify Motoko sync after upstream PR #6132#268

Open
marc0olo wants to merge 8 commits into
mainfrom
infra/motoko-sync-simplification
Open

infra: simplify Motoko sync after upstream PR #6132#268
marc0olo wants to merge 8 commits into
mainfrom
infra/motoko-sync-simplification

Conversation

@marc0olo
Copy link
Copy Markdown
Member

@marc0olo marc0olo commented May 20, 2026

Summary

Upstream caffeinelabs/motoko PRs #6132, #6139, and #6142 transformed doc/md/ into a Starlight-ready state, extracted the sidebar to a shareable doc/site/sidebar.mjs, and cleaned up stray content. This PR adopts those changes in the developer-docs sync pipeline, closing issue #265.

Changes

scripts/sync-motoko.sh — rewritten as plain rsync (~35 lines, down from ~180). All the complex bash transforms (frontmatter injection, .mdx.md conversion, numeric prefix stripping, slug-collision guards, H1 removal) are no-ops now that upstream handles them.

scripts/postprocess-motoko.mjs — seven steps removed (all no-ops after PR #6132). Remaining responsibilities:

  • Inline Changelog.md from the submodule at sync time. The upstream stub uses a build-time file= include that doesn't work inside the icp deploy recipe (submodule not in scope during npm run build). Inlining at sync time makes the file fully self-contained in docs/, and ensures em-dash replacement and link rewrites apply to the changelog content too.
  • Rewrite internetcomputer.org/docs/ and docs.internetcomputer.org/ links to internal paths.
  • Replace em-dashes (banned per ICP style guide).
  • Redirect stray core//base/ relative links to mops.one (safety net).
  • Inject sidebar.hidden: true for subdirectory index.md stubs (transitional fallback — now a no-op since upstream PR #6139 added hidden: true directly in the stubs).
  • Auto-generate sidebar-motoko.mjs from doc/site/sidebar.mjs in the submodule. Slugs and autogenerate.directory values get the languages/motoko/ prefix. If sidebar.mjs is absent (older submodule pin), generation is skipped and the committed file is used as-is.

sidebar-motoko.mjs — now auto-generated from the upstream doc/site/sidebar.mjs (added in caffeinelabs/motoko PR #6139). The upstream file is the single source of truth for Motoko navigation. To change the sidebar, edit doc/site/sidebar.mjs in caffeinelabs/motoko and run npm run sync:motoko.

ec.config.mjs + syntaxes/bnf.tmLanguage.json — register BNF TextMate grammar for Shiki so the 30 BNF blocks in language-manual.md no longer emit language-not-found warnings at build time.

.docs-plan/motoko-repo-sync-proposal.md — deleted. The proposal it tracked is now complete.

.sources/motoko — bumped to 1fc25e951 (master after PRs #6139 and #6142). Pre-release pin; once the Motoko team cuts a release tag, the automated sync-motoko.yml workflow will take over for future updates.

docs/languages/motoko/** — full resync from updated submodule.

Build

208 pages, no errors, no Shiki language-not-found warnings.

Sync recommendation

sync from caffeinelabs/motoko doc/md/ — after this PR, Motoko docs are maintained purely by running npm run sync:motoko against the pinned submodule. Navigation structure is driven by doc/site/sidebar.mjs upstream.

Testing

npm run build   # 208 pages, no errors
npm run dev     # verify sidebar: Motoko > Overview, Fundamentals (with correct group labels), ICP features, Reference
                # verify changelog page shows version entries (1.8.1, 1.7.0, ...)

Closes #265

Upstream caffeinelabs/motoko PR #6132 transformed doc/md/ into a
Starlight-ready state: clean filenames, sidebar.order frontmatter,
aside syntax normalized, mo:base→mo:core rewrites, numeric prefixes
removed, and index.md stubs added for subdirectory groups. This PR
adopts those changes in the developer-docs sync pipeline.

Changes:
- scripts/sync-motoko.sh: rewrite as plain rsync (~35 lines vs ~180).
  All the complex bash transforms (frontmatter injection, .mdx→.md
  conversion, numeric prefix stripping, slug collision guards) are
  no-ops now that upstream handles them.
- scripts/postprocess-motoko.mjs: remove seven steps that upstream
  now handles. Critical fix: the old script explicitly deleted the
  new index.md stubs PR #6132 added, breaking autogenerate. Also
  adds a general docs.internetcomputer.org/ prefix rule to handle
  links that upstream PR #6132 §5 rewrote from the retired portal
  format to the current docs.internetcomputer.org/ domain.
- sidebar-motoko.mjs: replace 130-line explicit page list with a
  section-level autogenerate config. Uses explicit section labels
  ("ICP features") to avoid the "Icp Features" label that plain
  autogenerate would produce for the icp-features/ directory.
- ec.config.mjs + syntaxes/bnf.tmLanguage.json: register BNF grammar
  for Shiki so the 30 BNF blocks in language-manual.md no longer
  emit language-not-found warnings.
- .docs-plan/motoko-repo-sync-proposal.md: delete (resolved by PR
  #6132; tracked in issue #265).
- .sources/motoko: bump to a9cc50f (master after PR #6132 merge;
  no release tag cut yet — this is a pre-release test pin).
- docs/languages/motoko/**: full resync from upstream (77 content
  files updated + 6 new index.md stubs for autogenerate groups).

Build: 208 pages, no errors.
@marc0olo marc0olo requested a review from a team as a code owner May 20, 2026 15:35
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 20, 2026

🤖 Here's your preview: https://lvbky-nyaaa-aaaam-ai7tq-cai.icp0.io

marc0olo added 2 commits May 20, 2026 18:04
The new autogenerate approach in sidebar-motoko.mjs broke the Motoko
sidebar in two ways:

1. Group labels showed raw directory names ("actors", "basic-syntax")
   instead of "Actors", "Basic syntax". In Starlight v0.38, groupFromDir
   uses the raw directory key as the label — it does not read index.md
   frontmatter to derive the label.

2. The index.md stubs (added by upstream PR #6132) appeared as duplicate
   page entries within the autogenerate groups ("Actors" page inside the
   "actors" group), and the root "Motoko" page showed as "Motoko" instead
   of "Overview".

Fix:
- sidebar-motoko.mjs: restore explicit items for fundamentals/ (matches
  the live site structure) and reference/ (needed for "Motoko changelog"
  label). icp-features/ keeps autogenerate — it is a flat list of pages
  with correct sidebar.order values and no subdirectories.
- postprocess-motoko.mjs: add sidebar.hidden: true to all subdirectory
  index.md stubs. This prevents them from appearing as stray page entries
  in autogenerate results. The root index.md (which has content) is
  excluded from this rule.
When .sources/motoko/doc/md/sidebar.mjs exists (added in upstream
caffeinelabs/motoko docs/sidebar-extraction), postprocess-motoko.mjs
now generates sidebar-motoko.mjs automatically by:
- importing the upstream sidebar array
- prefixing every slug with "languages/motoko/"
- prefixing every autogenerate.directory with "languages/motoko/"
- wrapping in the motokoSidebar export with label "Motoko"

When sidebar.mjs is absent (submodule on an older release), the step
is skipped and the committed sidebar-motoko.mjs is used unchanged.

Once the upstream PR merges and the submodule is bumped to a release
that includes sidebar.mjs, sidebar-motoko.mjs becomes a generated
artifact — no hand-editing required.
@marc0olo marc0olo marked this pull request as draft May 20, 2026 16:47
@marc0olo
Copy link
Copy Markdown
Member Author

marc0olo commented May 20, 2026

Waiting on: caffeinelabs/motoko docs/sidebar-extraction

This PR is blocked on an upstream change that eliminates the duplicated sidebar definition. Once that PR merges and a release tag is cut that includes it, this PR can be finalized.

What the upstream PR does (marc0olo/motokocaffeinelabs/motoko)

  • doc/site/sidebar.mjs (new): exports the Motoko sidebar as a single source of truth. Lives in doc/site/ alongside astro.config.mjs (site config, not content). Both the local preview site and the consumer site sync pipeline consume it.
  • doc/site/astro.config.mjs: imports sidebar from ./sidebar.mjs instead of hardcoding it inline. Eliminates the duplication.
  • doc/md/reference/changelog.md: adds sidebar.label: "Motoko changelog" so the autogenerate for reference/ produces the correct label on the consumer site without a special-case override.
  • doc/md/fundamentals/*/index.md (6 stubs): adds hidden: true upstream, making the postprocess-motoko.mjs injection step a no-op.

What happens in this PR once the upstream merges

  1. Bump .sources/motoko to the new release tag that includes doc/site/sidebar.mjs.
  2. Run npm run sync:motokopostprocess-motoko.mjs detects sidebar.mjs in the submodule at doc/site/sidebar.mjs and auto-generates sidebar-motoko.mjs. The explicit sidebar in the current sidebar-motoko.mjs is replaced by the generated version.
  3. Verify npm run build passes (208 pages, no errors).
  4. The hidden: true injection step in postprocess-motoko.mjs becomes a no-op (stubs already have it upstream).
  5. Mark as ready for review.

After this PR merges

sidebar-motoko.mjs is a generated artifact. Navigation changes in the Motoko docs are made once (in caffeinelabs/motoko doc/site/sidebar.mjs) and flow to the consumer site automatically on the next sync.

marc0olo added 3 commits May 20, 2026 18:59
sidebar.mjs will exist in doc/md/ once caffeinelabs/motoko#6139 merges.
postprocess-motoko.mjs reads it directly from the submodule path; it
should not be copied into docs/languages/motoko/ as a stray non-content
file.
…sion

sidebar.mjs moved to doc/site/ in caffeinelabs/motoko (site config,
not content). Update UPSTREAM_SIDEBAR path accordingly. The rsync
exclusion added in the previous commit is no longer needed since
doc/site/ is never part of the rsync source.
.sources/motoko bumped to b75b9aa91 (master after caffeinelabs/motoko
PR #6139: sidebar extraction to doc/site/sidebar.mjs). sidebar-motoko.mjs
is now auto-generated by postprocess-motoko.mjs from the upstream sidebar
definition. The hidden:true injection for index.md stubs and the Motoko
changelog label override are both no-ops — both handled upstream in #6139.
@marc0olo marc0olo marked this pull request as ready for review May 21, 2026 07:50
marc0olo added 2 commits May 21, 2026 10:33
The upstream changelog.md stub uses a build-time file include
(```md file=<motokoRoot>/Changelog.md```) that works locally but
produces an empty page in the icp deploy CI recipe (the submodule is
not in scope when the asset canister recipe runs npm run build).

Fix: expand the include in postprocess-motoko.mjs before walkAndProcess
so the file is fully self-contained in docs/ and em-dash replacement
and link rewrites apply to the inlined content too.
Picks up caffeinelabs/motoko PR #6142: removes the auth-gated
DFINITY Frontify logo that was accidentally left at the bottom of
actors-async.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

infra: follow-up actions after caffeinelabs/motoko#6132 merges

1 participant